Skip to content

Solver jobs#157

Closed
patricoferris wants to merge 2 commits intoocurrent:masterfrom
patricoferris:solver-jobs
Closed

Solver jobs#157
patricoferris wants to merge 2 commits intoocurrent:masterfrom
patricoferris:solver-jobs

Conversation

@patricoferris
Copy link
Contributor

@patricoferris patricoferris commented Jan 31, 2022

This PR builds on top of #156 (the first commit) and adds the following:

  • It vendors https://github.com/patricoferris/solver-service, which is the core logic of OCaml-CI's solver services in a separate library and executable. This is to hopefully future-proof expanding the use of a solver-service to other clients like opam-health-check/opam-repo-ci. The cluster workers are extended to support solving solver requests.
  • The scheduler is extended to support submitting custom job specifications.
  • The example adds a small solver job to get the dependencies of obuilder. It was also converted from OBuilder to Docker for portability (i.e. so it runs on my mac 🍎)

Here is the example pipeline:

Screenshot 2022-01-31 at 16 36 17

And the log at the time of writing for the solver job:

Solver job lob 
2022-01-31 16:36.44: New job: Custom job using linux-x86_64 in
                                [https://github.com/ocurrent/obuilder.git#refs/heads/master (df44de3a865129df9fe17d84c3006967746f3d0a)]

Custom job:

solve

2022-01-31 16:36.44: Using cache hint "https://github.com/ocurrent/obuilder.git"
2022-01-31 16:36.44: Connecting to build cluster...
2022-01-31 16:36.44: Waiting for resource in pool OCluster
2022-01-31 16:36.44: Waiting for worker...
2022-01-31 16:36.44: Got resource from pool OCluster
Building on build
All commits already cached
HEAD is now at df44de3 Merge pull request #97 from mtelvers/master
2022-01-31 16:36.45 [INFO] Solving for obuilder-spec.dev, obuilder.dev
2022-01-31 16:37.22 [INFO] os: found solution in 0.72 s
2022-01-31 16:37.22 [INFO] = os =
2022-01-31 16:37.22 [INFO] -> alcotest.1.5.0 alcotest-lwt.1.5.0 astring.0.8.5
                              base.v0.14.3 base-bigarray.base base-bytes.base
                              base-domains.base base-nnp.base
                              base-threads.base base-unix.base
                              bigarray-compat.1.1.0 biniou.1.2.1
                              cmdliner.1.0.4 conf-pkg-config.2 conf-sqlite3.1
                              cppo.1.6.8 csexp.1.5.1 cstruct.6.0.1
                              cstruct-lwt.6.0.1 dune.2.9.3
                              dune-configurator.2.9.3 easy-format.1.3.2
                              fmt.0.9.0 logs.0.7.0 lwt.5.5.0 mmap.1.1.0
                              num.1.4 obuilder.dev obuilder-spec.dev
                              ocaml.5.00.0 ocaml-compiler-libs.v0.12.4
                              ocaml-config.2 ocaml-syntax-shims.1.0.0
                              ocaml-variants.5.00.0+trunk ocamlbuild.0.14.0
                              ocamlfind.1.9.3 ocplib-endian.1.2
                              parsexp.v0.14.2 ppx_cstruct.6.0.1
                              ppx_derivers.1.2.1 ppx_deriving.5.2.1
                              ppx_sexp_conv.v0.14.3 ppxlib.0.25.0~5.00preview
                              re.1.10.3 result.1.5 seq.base sexplib.v0.14.0
                              sexplib0.v0.14.0 sha.1.15.1 sqlite3.5.1.0
                              stdlib-shims.0.3.0 tar.2.0.0 tar-unix.2.0.0
                              topkg.1.0.5 uchar.0.0.2 uutf.1.0.2 yojson.1.7.0
2022-01-31 16:37.22 [INFO] (valid since opam-repository commit f9f1b91b24a391866ce58902d1d7a588cb8009a6)
["Ok",[{"id":"os","packages":["alcotest.1.5.0","alcotest-lwt.1.5.0","astring.0.8.5","base.v0.14.3","base-bigarray.base","base-bytes.base","base-domains.base","base-nnp.base","base-threads.base","base-unix.base","bigarray-compat.1.1.0","biniou.1.2.1","cmdliner.1.0.4","conf-pkg-config.2","conf-sqlite3.1","cppo.1.6.8","csexp.1.5.1","cstruct.6.0.1","cstruct-lwt.6.0.1","dune.2.9.3","dune-configurator.2.9.3","easy-format.1.3.2","fmt.0.9.0","logs.0.7.0","lwt.5.5.0","mmap.1.1.0","num.1.4","obuilder.dev","obuilder-spec.dev","ocaml.5.00.0","ocaml-compiler-libs.v0.12.4","ocaml-config.2","ocaml-syntax-shims.1.0.0","ocaml-variants.5.00.0+trunk","ocamlbuild.0.14.0","ocamlfind.1.9.3","ocplib-endian.1.2","parsexp.v0.14.2","ppx_cstruct.6.0.1","ppx_derivers.1.2.1","ppx_deriving.5.2.1","ppx_sexp_conv.v0.14.3","ppxlib.0.25.0~5.00preview","re.1.10.3","result.1.5","seq.base","sexplib.v0.14.0","sexplib0.v0.14.0","sha.1.15.1","sqlite3.5.1.0","stdlib-shims.0.3.0","tar.2.0.0","tar-unix.2.0.0","topkg.1.0.5","uchar.0.0.2","uutf.1.0.2","yojson.1.7.0"],"commit":"f9f1b91b24a391866ce58902d1d7a588cb8009a6"}]]Job succeeded
2022-01-31 16:37.22: Job succeeded

talex5
talex5 reviewed Feb 1, 2022
View reviewed changes
Copy link
Contributor

@talex5 talex5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

But I think it would be better to keep OCluster as a generic build service (not specific to OCaml or CI stuff), and instead deploy the solver as its own service, connected to its own pool on the scheduler. Allowing this separation was the reason for using AnyPointer and generic jobs, rather than adding a specific solver job type.

@patricoferris
Copy link
Contributor Author

patricoferris commented Feb 1, 2022

Ah okay makes sense, thanks for taking a look 👍

I'll pull out the changes to the scheduler and put them on #156 though because at the moment it doesn't actually support custom job submission... (i.e. https://github.com/ocurrent/ocluster/pull/157/files#diff-bb8011a7ab51aad47afd6d179def81780180a05239312e01bb6c75a1e607f8b3)

@patricoferris
Copy link
Contributor Author

patricoferris commented Feb 1, 2022

Closing this PR now in favour of:

Thanks for the quick clarifications :))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@talex5 talex5 talex5 left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@patricoferris @talex5